The potential of large language models (LLMs) in specialized domains such as legal risk analysis remains underexplored. In response to growing interest in locally deploying open-source LLMs for legal tasks while preserving data confidentiality, this paper introduces ContractEval, the first benchmark to thoroughly evaluate whether open-source LLMs could match proprietary LLMs in identifying clause-level legal risks in commercial contracts. Using the Contract Understanding Atticus Dataset (CUAD), we assess 4 proprietary and 15 open-source LLMs. Our results highlight five key findings: (1) Proprietary models outperform open-source models in both correctness and output effectiveness, though some open-source models are competitive in certain specific dimensions. (2) Larger open-source models generally perform better, though the improvement slows down as models get bigger. (3) Reasoning ("thinking") mode improves output effectiveness but reduces correctness, likely due to over-complicating simpler tasks. (4) Open-source models generate "no related clause" responses more frequently even when relevant clauses are present. This suggests "laziness" in thinking or low confidence in extracting relevant content. (5) Model quantization speeds up inference but at the cost of performance drop, showing the tradeoff between efficiency and accuracy. These findings suggest that while most LLMs perform at a level comparable to junior legal assistants, open-source models require targeted fine-tuning to ensure correctness and effectiveness in high-stakes legal settings. ContractEval offers a solid benchmark to guide future development of legal-domain LLMs.

Despite an increasing academic interest in Legal NLP research over the last years, AI-assisted contract review, especially in languages other than English, has received little attention [KATZ 2023]. One major hurdle for that may be the scarcity of sufficient, annotated training data. Semantic annotations of legal texts can only be done by legal experts, resulting in high costs and a scarcity of publicly available datasets. The situation worsens when legal texts, such as employment contracts, include sensitive personal information. A partnership with a German law firm specializing in Economic Law now enables us to conduct more research in this area. As part of a collaborative project, we aim to design, implement, and evaluate a prototypical AIbased system for assisting in the review and correction of German employment contracts. To initiate our research efforts and encourage further investigations and experiments by other researchers, we release an anonymized and annotated dataset of clauses from German employment contracts (License: CC BY-NC 4.0), along with their respective legality and categorization labels. Additionally, we provide benchmarks for both open-and closed-source baseline models.

However, as of the current state of research, there appears to be a significant gap in exploratory and experimental studies specifically addressing the capabilities of Generative AI and Large Language Models (LLMs) in the context of determination and discovery of legal issues. Such studies would be instrumental in understanding how these advanced AI technologies manage the intricate task of accurately classifying and pinpointing legal matters, a domain traditionally reliant on the deep, contextual, and specialised knowledge of human legal experts. To address the identified gap in the research landscape, this study proposes an experimental and exploratory analysis of the performance of LLMs in the legal domain. The research aims to evaluate the capabilities of LLMs contrasting their performance against human legal practitioners on high volume real-world legal tasks. These types of high volume legal tasks are frequently outsourced or pushed to less experienced lawyers, and given the rapid advancements made by LLMs, raises the question of whether LLMs have achieved a level of legal comprehension that is comparable to the quality, accuracy and efficiency of Junior Lawyers or outsourced legal practitioners on such tasks.

Reading comprehension of legal text can be a particularly challenging task due to the length and complexity of legal clauses and a shortage of expert-annotated datasets. To address this challenge, we introduce the Merger Agreement Understanding Dataset (MAUD), an expert-annotated reading comprehension dataset based on the American Bar Association's 2021 Public Target Deal Points Study, with over 39,000 examples and over 47,000 total annotations. Our fine-tuned Transformer baselines show promising results, with models performing well above random on most questions. However, on a large subset of questions, there is still room for significant improvement. As the only expert-annotated merger agreement dataset, MAUD is valuable as a benchmark for both the legal profession and the NLP community.

The legal profession has a long history of keeping pace with technology as it advances. With the development and spread of artificial intelligence (AI), various professions have embraced its ability to automate tasks that people once did. This shift has caused anxiety among lawyers, who worry about losing their jobs to machines. But it is becoming clear that, as AI evolves, lawyers will find new and innovative ways to use it in their practices. AI is already used in some law firms to automate such tasks as contract review and discovery.

Contract lifecycle management (CLM), the method of managing a contract from initiation through award, compliance, and renewal, can be costly for companies. World Commerce and Contracting estimates the average cost of a simple contract at $6,900, rising to over $49,000 for more complex agreements. The opportunity is often worth the investment, but without close contract governance, businesses stand to lose up to 40% of a contract's value, a KPMG survey found. The tantalizing prospect of automating the contracting process has drawn a number of entrepreneurs to the space, including UnitedLex co-founder Ajay Agrawal. Agrawal's newest venture is SirionLabs, which comines AI technologies like natural language processing to import and organize contracts, negotiations, and contract review. Highlighting the investor interest in the segment, SirionLabs announced that it closed an $85 million Series D financing round led by Partners Group with participation from existing investors Sequoia Capital and Tiger Global.

Artificial Intelligence has struck almost every domain and results are getting accurate day by day. We present a study of how AI is automating law and legal domain and what can be some of the future applications which can help the legal industry to automate their process. In the past, contract review auditors had two tasks that appeared to be in direct conflict with each other: to perform a contract review quickly and to conduct a thorough contract review. To balance these competing ends, auditors take representative samples from contracts rather than working on the entire data community. It's no surprise then that accounting departments and corporate finance institutions are turning to cognitive technologies to help Deliver more value and improve the bottom line of the organization.

AI is a hot buzzword right now, but with buzz always comes a whole host of misconceptions about a technology's capabilities. There's considerable confusion about what artificial intelligence can do and widespread misinformation about how it works, particularly in the area of managing legal contracts and if AI will replace lawyers. Onit recently hosted a webinar to debunk these common myths. Nick Whitehouse, General Manager of Onit's AI Center of Excellence, and Jean Yang, Vice President of Onit's AI Center of Excellence, dispelled common misconceptions about everything from will AI replace lawyers to who can benefit from AI. The goal is to help legal professionals decipher marketing-speak to determine what's genuinely AI and what's just software.

Summary: Some industries are a clear slam-dunk for AI/ML applications and some less so. The legal, regulatory, and compliance businesses (law firms, internal legal departments, and the contract review and regulatory compliance departments of heavily regulated industries) fall in this last category. This is a review of seven companies found by TopBots to be successful; pointing to opportunities others can follow. Remember just a few years ago when we were looking forward to now or a little beyond and imagining what applications AI/ML would have in different industries. Some of those prognostications were slam dunks as they applied to customer propensity or using machine vision to count whatever widgets you were interested in.

ThoughtRiver, a London-based legaltech startup that's applying AI to speed up contract pre-screening, has announced a $10 million Series A round of funding led by Octopus Ventures. Existing seed investors Crane, Local Globe, Entrée Capital, Syndicate Room, and angel investor Duncan Painter also participated in the round. The UK startup is one of a number applying AI to automate work that would otherwise be done by legal professions with the aim of boosting operational efficiency. Other startups playing in the space include the likes of Kira Systems, LawGeex and Luminance to name a few. ThoughtRiver argues it has a different focus vs the majority of contract view companies because it's focusing on pre-signature contracts -- with the aim of making securing a deal faster. "Almost all others are just employed to pull data from existing contracts.